A Fine-Grained, Dynamic Load Distribution Model for Parallel Stream Processing
نویسنده
چکیده
Our goal is to address the unique characteristics and limitations of emerging large-scale commodity clusters to leverage their potential for the parallel processing of multidimensional data streams. To this end, we describe a new distributed stream processing model that integrates data and task parallelism by partitioning workloads into selfdescribing chunks that are dynamically assigned to available computing resources. We adapt the degree and means of processing parallelism using simulation-driven heuristic search algorithms with underlying incremental bin-packing techniques to direct chunk assignments, facilitating adaptation to fluctuating resource availability and workloads characteristics. Our experimental study quantifies the potential yields of our approach under a variety of workload and computing configurations.
منابع مشابه
Parleda: a Library for Parallel Processing in Computational Geometry Applications
ParLeda is a software library that provides the basic primitives needed for parallel implementation of computational geometry applications. It can also be used in implementing a parallel application that uses geometric data structures. The parallel model that we use is based on a new heterogeneous parallel model named HBSP, which is based on BSP and is introduced here. ParLeda uses two main lib...
متن کاملAdaptive Load Diffusion for Stream Joins
Data stream processing has become increasingly important as many emerging applications call for sophisticated realtime processing over data streams, such as stock trading surveillance, network traffic monitoring, and sensor data analysis. Stream joins are among the most important stream processing operations, which can be used to detect linkages and correlations between different data streams. ...
متن کاملEvolution of texture in an ultrafine and nano grained magnesium alloy
The evolution of texture was discussed during the formation of ultra-fine and nano grains in a magnesium alloy severely deformed through accumulative back extrusion (ABE). The microstructure and texture obtained after applying multiple deformation passes at temperatures of 100 and 250°C were characterized. The results showed that after single ABE pass at 100°C an ultrafine/nano grained microstr...
متن کاملParallel branch and bound on fine-grained hypercube multiprocessors
In this paper, we study parallel branch and bound on fine grained hypercube multiprocessors. Each processor in a fine grained system has only a very small amount of memory available, Therefore, current parallel branch and bound methods for coarse grained systems (s 1000 nodes) can not be applied, since all these methods assume that every processor stores the path from the node it is currently p...
متن کاملEfficient Migration of Very Large Distributed State for Scalable Stream Processing
Any scalable stream data processing engine must handle the dynamic nature of data streams and it must quickly react to every fluctuation in the data rate. Many systems successfully address data rate spikes through resource elasticity and dynamic load balancing. The main challenge is the presence of stateful operators because their internal, mutable state must be scaled out while assuring fault-...
متن کامل